DOCOflENT RESOIE 



ED 127 471 

AOTHOE 
TITLE 

ISSTITOTION 

PUB DATE 
NOTE 

BDHS PRICE 
DESCHIPTOHS 



CE 007 728 

Hoyty Kenneth 3. 

Perspectives on the Problem of Evaluation in Caree: 
Education . Monographs on Career Education. 
Office of Career Education (DHEH/OE) , Rasuington^ 
D. C. 
[76] 

aap. 

MF-$0.83 HC-$2.06 Plus Postage. 
Administrative Problems; *Career Education; 
Conference Reports; ^Educational Assessment; 
♦Educational Objectives; Evaluation Criteria; 
♦Evaluation Methods; ♦Program Evaluation; Research 
Problems 



ABSTRACT 

This monograph summarizes the results of two 
mini-conference discussions on the general topic, evaluation of 
career education. It addresses three major points: (1) Problems 
involved in evaluation of career education, i.«., instrumentation, 
expectation, concept implementation, and perceived personal benefit, 
(2) examples of evaluation approaches currently utilized in career 
education by 12 professionals with direct responsibility for 
evaluation, and (3) learner outcomes (which are stated as goals and 
include consultant recommendations for each goal) appropriate for use 
in evaluating career education. Appendices list names and addresses 
of participants in the two mini-conferences. (SH) 
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PREFACE 



During 1975-76, the Office of Career Education (OCE) 
conducted, through provisions of a grant made to The 
Center for Vocational Education, The Ohio State Univer- 
sity, a series of 27 "mini-conferences'' covering a wide 
range of career education topics. Two of these "mini- 
conferences", one involving expert practitioners and 
the other involving general consultants, were devoted 
to the general topic "Evaluation of Career Education". 
It is the results of deliberations from these two con- 
ferences that are summarized here. The general notes, 
on which this monograph is based, are contained in the 
technical report by Dr. Richard Miguel, Project Direc^ 
tor for the Office of Education (OE) grant concerned 
with the 1975-76 "mini-conferences". 

Many examples of evaluation practices and results 
are contained in the files of OE*s Office of Career 
Education. These include unsolicited examples from 
current local career eaucati^^n projects as well as 
formal evaluations for each of the 1975-76 OCE funded 
career education projects. Such documents are avail- 
able for study by those visiting the Office of Career 
Education, but are not included as part of this mono- 
graph. Instead, an attempt has been made here to 
limit discussion to topics discussed during these two 
"mit.i-conferences". It is hoped that such a discussion 
will be helpful no those concerned with the problem of 
evaluation of career education. 

Tae list of participants in these two "mini-confer- 
ences" is incli'ded in the Appendix* While each listed 
participant m^de significant contributions, none can 
or should be held accountable for the specific content 
of this monograph. The examples cf specific evaluation 
practices attributed to particular participants can, in 
each instance, be substantially expanded by those who 
wish to make contact with such participants. 



PERSPECTIVES ON THE PROBLEM OF EVALUATION 
IN CAREER EDUCATION 

INTRODUCTION 

The age of accountability has definitely come to 
American education. In general, the accountability 
emphasis has called on all parts of American education 
to answer two basic questions: (a) Vhat have you done? 
(this is the question of pKOCU^) and (b) What benefits 
have resulted from your actions? x'this is tl^e q^iestion 
of (XloducX) » Each of the long-established programs and 
pjrts of our educational system are currently faced 
with providing definitive answers to these two questions. 
The general thrust has been one aimed at moving away from 
the trend of xaaking education cost ^re and more money, 
and toward a direction of making education more cost- 
effective. As a general trend, it is one that should be 
welcomed, rather than resisted, by all educators. 

Ideally, of course, the question cr hotjC good a par- 
ticular educational practice is will be -sked only after 
the question of vOhoX iX t6 has been clearly answered. 
The impact of the accountability issue in education, at 
the present time, has not permitted this ideal situation 
to prevail. This is especially true with any proposals 
for rtCM^ approaches to education — such a? career educa- 
tion. To the extent that a suggested new proposal calls 
for basic changz in any part of education, that change is 
certain to meet with a certain amount of resistance on 
the part of those being asked to change. When, as with 
career education, change is called for in the tnti/it 
^(/A-te/yi of American education, the degree of resistance 
to change becomes very great indeed. One of the easiest, 
and currently most popular, ways of expressing resistance 
to change is to note that the proposed new process has 
not, as yet, been subjected to sufficiently rigorous 
evaluation. The age of accountability has increased the 
ude of this strategy for resisting change. 

If carried to an extreme, the general call for 
accountability can effecti\'ely hinder the introduction 
of any new educational concept. That is, the expected 
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benefits f ror education , alMSt without exception , are 
stated in long-terin goals as veil as in short-term goals. 
The long-terni goals, typically, are stated in terins that 
reflect expected behaviors of pupils once they have be- 
come adults and left the fornal syster of education. 
Such goals, ther^^fore, are ones t!iat defy evaluation 
prior to the time s generat ion of pupils has passed 
through the educational systex and assumed adult roles. 
When evidence of effectiveness ir attaining such goals 
are demanded prior to the time a new practice is per- 
mitted to be tried, there is no way that such demands 
can be net. Tn the case of career education, which 
assumes efforts beginning in the ^arly elementary school 
years and continuing through ti e ':?ntire system of formal 
education, the attainment of long-run goals must neces- 
sarily be delayed for a number of years. 

This m*akes it doubly important that careful attention 
be paid to both process goals and to short-term product 
goals of career education. This need has been recog- 
nized by career education advocates from the beginning. 
Here, an attempt will be made to summarize thoughts and 
recomm.endations of a number of career education evalua- 
tion experts related to this need. At present, both 
our knowledge and our practices remain too incomplete 
as to result in any simple, "cookb,:>ok'* solutions for 
practitioners. As a result, this monograph has contented 
itself with addressing three m^ijor topics: (a) Problems 
Involved in evaluation of career cdacdt ion; (b) Examriltis 
of evaluation approaches currently utilized in career 
education; and (c) Learner outcomes appropriate for use 
in evaluacinp career education. 

PROBLEMS TN EVALUATION OF CAREER LDUCATTON 

It is ersler to state, than to solve, problems In 
alm.ost any area of endeavor. This general maxim was 
demonstrated repeatedly in the two "mini-conrerences" 
forming the basis for this monograph. A second maxim 
can be stated in the following way: 7^: Lb ^po66<blCL to 
^tclvc a p-xcb^cm un?(i^6 itt dlrDcyiticn^t can 6e clzarltj 
defined. A considerably amount of "mini-conference" 
time was devoted to discussion centered around this 
maxim*. Results of those discussions are reported in 
this section. 
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THE PROBIrIM OF INSTRUMENTATION 



Perhaps the most frequently mentioned problem 
raised by participants was that of appropriate instru- 
centation for evaluation of career education. Clear 
consensus appeared to be present that such instrumen- 
tation does not, at this tine, exist in a form that 
adequately covers all of the problems involved In 
evaluating career education. The prime difficulty 
appears to sten from the nature of the career education 
concept . 

For exanple, career education, from the outset, has 
been pictured as an effort ained at changing both the 
attitudes and the actions of educators, members of the 
business-labor-industry connunlty, and members of the 
home and far;ily structure. It is clear that pzcplc. 
change LEADING TO pXogKnn change is what we seek. 
Before we can e>q^ect program changes to occur to a 
decree that ixxkes then susceptible to evaluation, we 
must first assure ourselves that people change — i.e., 
chanj;cs in the attitiidilt of people — has occurred. 

The kinds of attitude change career education seeks 
cover a number of areas. Part of those attitudes have 
to do with recognizing (and believing) basic facts con- 
cerning changing relationships between education and 
vork. A second part has to do with attitudes tou'ard 
Involving the broader community in the teaching/learning 
process. A third part relates to attitudes toward the 
goal of education, as preparation for work, and the 
Importance of this goal among the basic goals of 
Arr.erican education. A fourth part relates to the per- 
ceived importance of bringing educational goals into 
the teaching/learning process as a source of educational 
motivation — and tht importance of motivation itself in 
the teaching/ leanilrg process. 

To date, very few instruments have been developed for 
assessing the kinds of attitudin^l changes called for by 
career education. Those that have been developed have 
often ignored the basic problem of validating such atti- 
tudlnal measures. The difficulty stems, in large part, 
from the fact that the kinds of attitudes being espoused 
by career education are those which, in a societal sense 
are considered desirable by large numbers of versons but 
for a variety of reasons^ are internalized by relatively 
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few. In part, uhe difficulty stems from the fact that, 
even when the attitude is clearly held, its implications 
for changes in actyion on the part of those who hold it 
often remain unclear. Without, in any sense, attempting 
to construct an attitude scale here, the following 
exaJ5»'ples of attitudes career education seeks to see 
inf malized, as part of the pzoplz ckangz process, 
include the following: 

1. The full range of both educational and occupa- 
tional opportunities must be made available for 
consideration by both ralnDrity persons and by women, 
as well as by white males. 

2. It is inrportant for today* s youth to understand 
and to act on the concept of lifelong learning. 

3. Adaptability skills — the abi'^.ity to change with 
change — are needed by today's youth fully as much as 
are entry level skills. 

4. '*Vork'* 1^ a humanistic word representing the 
basic human need of all human beings to do — to accom- 
plish — to achieve. As such, it can be easily dis- 
tinguished from the word "labor". 

5. The correlation between number oi years spent in 
schooling and economic rewards, while still positive, 
Is declining. 

6. The changing nature of the home and family struc- 
ture in America ir*akes it imperative that educators 
accept new kinds of responsibilities in imparting the 
concept of work in the home and family structure to 
youth, 

7. The goal of education, as preparation for work, 
must extend beyond simply the world of paid employ- 
ment and include both the concept of volunteerism and 
the productive use of leisure time. 

8. General education and the liberal arts are as im^- 
portant as are vocational educatiori and profesoionfll 
specialization curricula in iseetiug the goal of 
education, as preparation for work, in 'joday's society. 



9. Students can and do learn in niore ways than from 
bocks, in more places than in classrooms, and from 
r.ore people than only certified teachers. 

10. T}\e "world of schocline," and the ''world of paid 
erploycflent " rust be neshed in vays that will help 
youth TTiiike an effective transition from one of these 
"worlds" to the other. 

11. The rost f VJ^cc vocation^sl skills ,ire the basic 
acadt'ric skills . 

12. Work values, like other parts of one's personal 
value syste-. , are highly influenced by events taking 
pI.Hce durin:"^, the elementary school years. 

13. Sinc^ there are severe liriits to what one can 
l<?arn ahou*: work simply through reading, work experi- 
ence should become a general educational methodology 
rather tli.in a special kind of educational program. 

[iducational reform cannot be accomplished chrough 
prograri "add ons", but, rather, through changes 
lnfust?d into the existing: structure. 

15. To help students understand the career implica- 
tions of subject natter can be and is a powerful 
source of educational motivation for many students. 

16. Any person whc/ deserves to be called a "teacher", 
as opposed to an 'instructor*', must be concerned 
about motivating students to learn subject ai^tter In 
addition to being concerned about the subject matter 
to be Imparted. 



The few illustrative examples presented abo/e should 
make clear some of the major problems concerned with 
assessing attitude change in the evaluation of career 
education. First, the kinds of attitudes Illustrated 
here are ones that many of today's educators would tend 
to mark '*true** If presented In the form of an attltudl- 
nal measure. Second, because an Item may be marked true 
in no way necessrarlly means that the person marking It 
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fully understands the attitudinal statement itself. 
(As a matter of fact, the acquisition of such real 
understandings is one of career education's najor 
challenges at tae present tine.) Third, even when a 
given attitude is fully understood, there is no neces- 
sary assurance that it will become an internalized 
action commitment on the part of the person involved. 
Finally, even when such internal comitment to action is 
present, the person often can see the constraints imped- 
ing action more clearly than he/she can nee th.e routes 
to effective implementation actions. For all of these 
reasons, the valid assessment of attitudinal changes, so 
essential to actions educators may choose to taVe in im- 
plementing the career education concept, remains a criti- 
cal problem facing those irteresteu in the evaluation of 
career education. 

Uien the focus of attention is switclied from a concern 
v^brut attitude change among educators to attitudinal and 
behavioral changes among students , tlie instrumentation 
problem becomes even more severe. Career education ad- 
vocates have, for example, placed great stress on the 
importance of helping students acquire a personally 
meaningful set of vorV values* At this point in time, 
there is far from universal agreement with respect to 
the '*ange and specific definitions appropriate for utiv 
in describing, let alone measuring,, what we mean by 
"work values'*. Even when one is satisfied with a given 
description of wliat is regarded as a complete set of 
work values, there is obviously no way of saying one 
student's work values are "better*' chan another's. Tlius, 
evaluation attempts in this area must be lirited to try- 
ing to answer such questions as: (a) To what extent is 
the stude^'t aware of the full range of existing work 
values?; (b) To what degree has the student decided on 
those work values most im.portant to him/her?; and (c) 
What behaviors has the student demonstrated illustrating 
his/her attempts to implement work values? To answer 
any of these questions demands the presence of a mensure 
of work values that possesses adeq^-ate reliability and 
validity. It appears that much rem^iins to be done 
before we can ascribe such qualities to currently 
existing measures cf work values. 

Similar problems exist when such topics as "career 
decision-making skills", "employability skills", and 
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''good work habits" are raised. Efforts to develop and 
validate good measures in areas such as these, while 
currently the object of considerable attention, have yet 
to prod .ce anything approaching universally acceptable 
products. Even when tlie topic is listed as "basic 
acadenic skills*', there is wide diversity of opinion 
rer,ardinp; what would b? considered acceptable measures 
for use in deternining the extent to which students have 
acquired such skills- Sone of the current attempts to 
solve these kinds of 'instrumentation problems will be 
discussed in the last section of this monograph when 
the viability of OF's learner outcor.es for career 
education arc considered. Suffice it to say here th-t 
probleT.^^ Jo exist and lliey are serious in nature. 



r::\Pr:cTATii)Ns or career edi-cmion: 

REFORM VS. STI'I'ENT BENEFITS 

Great confusion continues to exist with respect to 
tl^e r,oal^ ana expectations of the career education 
effort. A prime source of confusion is clearly seen 
wlien one considers the question of educational reform 
vs, the question of student benefits to be derived 
(hopefully) as a result of educational reform. In 
brief, it is the classical problem of evaluation that 
demands on.- (a) specify clearly the treatment to be 
applied and (b) verify the ext<?nt to which the treat- 
ment has been applied prior to (c) assessing the bene- 
fits that result from application of the treatnient. 
Too many current attempts to evaluate career education 
have concentrated their primary attention on Step (c) 
while almost completely ignoring Steps (a) and (b). 
While the reasons for this are readily understandable, 
they are certainly not acceptable. 

Career education has been pictured as an attempt to 
XClfiCVi the entire system of American education in ways 
that will bring a more p^cpCA (note — not an ZX.clu6ivz) 
emphasis to education, as preparation for work, among 
the several basic goals of education. It is assumed 
iThat , if this reform can be accomplished, both student 
and societal benefits will result. Prior to testing 
that assumption, it is obviously first essential to 
specify the nature of tlie reform being sought and then 
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to assess the extent to which it has taken place. The 
problen is further complicated when one realizes that 
we are talking simultaneous;/ about using a different 
basic methodological approach to education (education 
bif rvza^it caiCCt6) and emphasizing specific learner 
outcomes related to career developr^ent (eduvation <>t 

Insofar as career education represents an effort 
aimed at reform of the entire ^^y^tcn of formal educa- 
tion, then it is obvious that evaluacin^ improvements 
in education — using criteria associated with any of its 
bcLtic goals — must be considered one means of evaluating 
the impact of career education. This is a prime reason 
why the criterion of "increases in basic academic skills" 
is listed as the first of the career education "learner 
outcomes" in the OE policy paper. An hit-.cd^jr,Ucn Tc 
Ca^iZtr Education. In order to specify the career educa- 
tion "treatment", it is essential that we be able to 
describe the basic nature of the kinds of educational 
reform being sought in ways that are surceptible to 
measurement. Since the career education effort is pic- 
tured as covering the entire 5i/6tCi^ of education (from 
the elementary school through the graduate college) and 
since the call for reform extends to all of the existing 
kinds of educational prcgnnrr^ and ipCCiaC ticb (e.g,. 
teachers, counselors, administrators, etc.). It is 
obvious that specification of specific kinds of changes 
being sought, in terms suitable for clear evaluations, 
vlll entail development of a very sizeable matrix of 
process variables. TVie development of sucli a matrix — 
and obcaining consensus on its components — is a task 
that, as yet, has not been completed. In spite of this, 
the major, generic kinds of changes being sought are 
ones on which wide, general agreement now appears to be 
present. Such generic kinds of changes include: 

1. A change Oioat/ from strict dependence on didactic 
instruction and f'.Circi\db a more activity-oriented, 
experiential approach to learning. 

2. A change aL<m{ from a single approach to instruc- 
tion and towa^td the use of a variety of methods for 
use by both teachers and students in the teaching/ 
learning process. 

1 L 
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3. A change aioay from depending on the teacher as 
the sole instructional resource available for helping 
students learn and tomjid the use of a wide variety 
of comniunity resource persons in the classroom. 

4. A change Oioaij from limiting instruction to formal 
classroom settings and toiAXXJud greater use of the 
coTTnnunity as a learning resource center* 

3. A change Oi^yiij from emphasizing to students how 
t'.ey have failed and toiOd^A emphasizing to students 
how they have succeeded. 

6. A change OioaiJ from worahiping time as a criterion 
""^r certifying educational accom^illshment and tcnXioAd 
rhe use of performance evaluation. 

7. A change mau from a great dependence on commer- 
cially prepared materials as instructional aids and 
tc'xcird the use of "homemade" materials developed by 
teachers, students, and community resourcr persins. 

8. A change ai^nui from emphasizing only the acquisi- 
tion of specific entry-lf>vel occupational skills and 
toioa-td adding an emphasis on helping students acquire 
adaptability skills required to change with occupa- 
tional change. 

9. A change aicay from emphasizing the goal of educa- 
tion, as preparation for work, only in terms of che 
occupational society and tvu^aid adding an emphasis 

cn education for productive use of leisure time. 

10. A change aioatj from vievixng work experience as a 
kind of educational program and tCiCOAd using work 
experience as a general educational methodology. 

11. A change awaij from viewing the prime purpose of 
-ducation as preparation ^or still more education and 
-^..'a/-.^ emphasizing the need to show students how what 
they lea n can be used outside the structure of formal 
education. 

12 
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12. A change CUMy from relegating the goal of educa- 
tion, as preparation for work, to only a portion of 
educational programs and toiWid finding ways of 
ciaking this a meaningful goal of all who teach and 
all who learn* 

13. A change oioay from strict dependence on profes- 
sional counselors for carrying out the career 
guidance function and tdCCVid the addition of 
teachers, parents, and community resource persons in 
this effort. 



In all of these ways, an educational institution that 
professes to have implemented a career education effort 
should be able to demonstrate the presence of activities 
in a form susceptible to measurement — and so to evalua- 
tion* It will be noted that the first seven of the 
thirteen changes listed above could be applicable to 
those concerned with any of the basic goals of educa- 
tion, while the last six are much more specifically 
related to only one goal — namely^ education as prepara- 
tion for work. This is in keeping with career educa- 
tion's basic tenet that the reforms being sought should 
aid, rather than detract from, efforts to meet all of 
the basic goals of American education. Career education 
asks that these first seven changes be carried out by 
means of a career emphasis. If such an emphasis can 
result in these kinds of changes, career education 
advocates have claimed that all basic goals of American 
education can be better met. This, too, represents a 
claim that should be susceptible to evaluation. 

When one moves from an "education by means of 
careers" emphasis to an "education in careers" emphasis, 
then product evaluation — stated in terms of learner out- 
comes — ^becomes more appropriate for use than the process 
oriented approach to evaluation discussed here. A dis- 
cxxssion of such student learner outcomes constitutes the 
last part of this monograph. Here, the point to be 
emphasized is that process evaluation, oriented around 
the basic kinds of changes being sought in the system of 
education, can be viewed as properly preceding product 
evalxiation as measured "by direct learner outcomes. In 
the long ran, both are irxportant and necessary. 
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THE PROBLEM OF EVALUATING 
IMPLEMENTATION OF A CONCEPT 

Career education has consistently sought to avoid 
becotaing yet another "program" to be added to all of 
those now existing In American education. Instead, 
It has sought to be regarded as a concept to be imple- 
mented throughout all of the various kinds of existing 
programs — and thus truly being a vehicle for reform 
of the entire educational ^y^tcm. Instead of asking 
for addition of a new course called "career education". 
It has asked for a "careers" emphasis to be Infused In 
all existing courses. Instead of asking for a new 
"career education specialist" to be added at the 
building level. It has asked for all existing educa-- 
tors to embrace and Implement the goal of education as 
preparation for work as part of their responsibilities 
to students. Thus, when one looks for a person, a 
course, or a special kind of building to use as a 
starting point in the evaluation of career education, 
none is found. Instead, career education's success, 
or lack of success, is to be found in looking at all 
educators, all courses, and all buildings in the system. 
Even beyond the formal education system Itself, career 
education has pictured itself as a collaborative that 
also involves both the business-labor- Industry commu- 
nity ar*d the home/f amily structure. These segments of 
society, too, must become involved when ore seeks to 
answer the questions of accountability in implementa- 
tion of the career education concept. 

Just as rhe career education effort hopefully makes 
positive con trrlbut ions to all other basic goals of 
education, so, too, do efforts aic^d at emphasizing 
such other goals hold positive potential for contribut- 
ing to the effectiveness of the career education effort. 
It is a two-edged svord and cuts both ways. For example, 
educators interested in promoting the basic goal of good 
citizenship m-y seek, as a vehicle for doing so, greater 
community interaction with the formal system of educa- 
tion. To the extent they are successful, the potential 
for increasing the efficacy of the career education 
effort is enhanced. Another segment of educators may 
express, as a primary interest, a concern for reduction 
of occupational sex role stereotyping in educational 
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materials and in career guidance services. This effort, 
too, holds positive potential for helping attain the 
objectives of career education, Withing the broader 
community, many kinds of activities are now operating — 
and did for several years prior to the time the words 
"career education'* were invented — aimed at helping youth 
in the career awareness, career exploration, and career 
decision-making processes. These include such organiza- 
tions as Junior Achievement, Explorer Scouts, Girl Scouts 
of the U.S,A.» the Chamber of Commerce, the National 
Alliance of Businessmen, the National Council of Business 
and Professional Women's Clubs, the National Council of 
Churches, and such major industrial corporations as the 
American Telephone and Telegraph Company, the General 
Electric Company, General Motors Corporation, and the 
American Cyanamid Company. There is no way in which 
one could pretend that activities of such groups do not 
contribute — or hold potential for contributing — to the 
success of the career education effort. 

Evaluation of the efficacy of career education, ii a 
given community, must, therefore, represent a combina- 
tion of two major kinds of process evaluation along with 
product evaluation based on soine set of leara^ir outcomes. 
One kind of process evaluation is required for each of 
the many educational and cotamunity segments called for 
in the collaborative process kncyvn as career education. 
The second kind of process evaluation must center around 
ways in which the various segments are brought together 
and interact together in ways that hold positive poten- 
tial for contributing to desired learner outcomes for 
career education. 

It is relatively easy to determine the extent to 
which the various segments — and people involved in each 
segment — are engaging in activities that appear to hold 
potential for the positive implementation of career 
education. It is, at the sane time, very difficult in- 
deed to look at positive results — in terms of learner 
outcomes — and specify which segment of education and /or 
of the broader community — made that outcome successful 
or unsuccessful. If career education ccmtinues to be 
viewed as a concept wiiich, when applied in action, can 
lead to both educational reform and to a series of 
direct student benefits, ve will continue to be in a 
position where the total success of the effort can be 
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evaluated, but great difficulty will be encountered 
when either praise or blame Is to be assigned to a 
particular segment. Evaluation would be much easier 
If career education were a program, but It Is not. 

THE PROBLEM OF "WHAT'S IN IT FOR ME?" 

It would be Ideal If everyone's personal values were 
so altruistic In nature that simply seeing positive 
student outcomes was a sufficient source of motivation 
for a particular person to continue his/her efforts. 
Unfortunately, we do not live In an "ideal world", but 
rather In a world of hiiman beings whose motivations, 
while sometimes based on altruism, are almost always 
also based on some kind of self-serving interest. Of 
all the "actors" In career education. It may well be 
true that the only ones who will find student benefits 
alone a sufficient reason for continuing their enthu- 
siastic Involvement In the effort are parents. For 
most other persons, career education efforts, no matter 
how important or desirable they may appear to be, are 
not seen as the primary reason why that person is 
eniployed. That is, the teacher's main job is imparting 
substantive content to students; the counselor's main 
job is improving student self-understanding and 
decision-making; the administrator's main job is 
administration; and the businessperson's main job is 
the particular business or occupation in which he/she 
is engaged. Yet, career education asks each of these 
persons to play a key and crucial role in the imple- 
nentation of career education. For each to volunteer 
to play such a role in an enthusiastic manner, and on 
a continuing basis, demands that they see some benefits 
accruing to themselves as well as to students. Such 
perceived benefits become, in effect, another set of 
criteria by which each evaluates the efficacy of career 
education — and so r^eir personal commitment to the 
concept. 

Classroom teachers, at the K-12 level, have pictured 
such benefits to themselves as including: (a) a way of 
i:uDti\'ating students to learn subject matter; (b) a way 
for teachers, as veil as students, to learn nev and 
valuable things about occupations and about their 
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comnunlty; (c) a way ot finding and utilizing community 
resources so that the teacher does not have to depend 
completely on his/her knowledge in order to present 
Information to students; (d) a way of helping teachers 
better understand their students; and (e) a way of 
introducing sufficient variety into the teaching/ 
learning process so that teaching becomes more 
Interesting for the teacher. 

Counselors have seen career education as a vehicle 
for: (a) helping to attain the goalo of career guidance 
(which the counselor recognizes he/she cannot do alone); 
(b) gaining greater acceptance for counselors among 
teachers and community persons; and (c) gaining more 
community support for the career development process. 

Members of the business-labor-industr^ community 
have viewed career education as holding positive 
potential for: (a) gaining greater understanding and 
acceptance of the American economic system; (b) pro- 
ducing students who, when they leave the educational 
system, are better prepared to enter and become suc- 
cessful in the world of paid emplo>Tnent; and (c) re- 
storing a positive attitude toward work on the part of 
both youth and adults in the cominunity. 

School administrators and school board members 
have viewed career education as an effort which, if 
successful, will: (a) serve as a vehicle for improving 
basic academic skills of students; (b) provide a means 
of making education more cost-effective through greater 
utilization of community resources; and (c) gain greater 
coCTunity understanding and support for education, thus 
making it easier to gain financial support for education. 

Vocational educators have viewed career education 
as a laeans for gaining greater cotmunity understanding 
and support for their efforts as well as a neans of 
attracting ziore able students. Liberal arts educators, 
on the other hand, have viewed career education as a 
rationale for justifying the key role the liberal arts 
play in providing students with the adaptability skills 
needed to change with changes in the broader society 
as well as a means of helping prepare students for 
productive use of leisure time. 

It is neither proper nor feasible to build formal 
evaluaticyns of the career education effort arotmd the 
vide variety of hopes and expectations listed above. 
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Yet, in an informal vay, there seems little doubt but 
that particular individuals will be making their own 
informal evaluations of career education in accordance 
with some of these expectations. Those persons interes- 
ted in and concerned about the importance of evaluating 
career education cannot ignore such expectations — even 
though they will seldom find opportunity to include them 
in formal evaluation designs. It may well be that, be- 
cause career education seeks to be infused into all 
programs rather than added on as a new, separate pro- 
gram, because it does not seek new kinds of specialists 
at the building level, and because it does not ask for 
large amounts of dollars, the long-run future* of career 
education may, operationally, depend relatively more on 
these kinds of personalized, informal criteria than the 
more forrial evaluation efforts based strictly on either 
process or product. 



EXAMPLES OF APPROACHES TO 

E\\\LUATION IN CAREER EDUCATION 

The first of the two "srini-conferences" on "Evalua- 
tion of Career Education" contained Invited participants^ 
each of whom had direct responsibility for evaluation of 
career education in a particular cominunitv or in a 
particular State. In addition to providing valuable 
consultative assistance with reference to the kinds of 
probleins outlined in the preceding liection, each also 
volunteered a short description of their current efforts* 
This section is devoted to a sinnmary of those descrip- 
tions* Interested readers can obtain much more complete 
descriptions by contacting the individuals mentioned 
here • 



DR. FRANK RAPLEY 

The Jefferson County, Kentucky, career education 
project, funded during FY- 1975 as a demonstration pro- 
ject aimed at providing career education to minority 
and lov-income youth, operates at the K-12 level in 
Louisville, Kentucky, and in the rest of the coxmty. 
Dr* Rapley is the person vith prirse responsibility for 
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designing and carrying out evaluation of this career 
education effort. 

While concerned with both process and product 
evaluation, initial attention was centered around a 
fXXOCU^ approach to evaluation. Six major objectives 
of this project were first specified. Each objective 
was then sub-divided into a number of state- 
tnen*:s, and each mission statement further jjub-divided 
into a number of task analysis statements. For each 
task analysis statement, persons responsible for per- 
forming the task were identified and a time line con- 
structed indicating when that particular task was due 
to be completed. (The basic model was one they had 
used previously In a drug education project.) 

Each quarter, progress reports are prepared con- 
sisting of: (a) self reports of persons charged with 
performing each task; and (b) interview data collected 
from at least a sairple of intended recipients of the 
activity. In addition, quarterly reports, known as 
"Documentation Reports" are prepared, in computerized 
form, consisting of three parts: (a) Action Plan 
Reports; (b) Accomplishment Reports; and (c) Change 
Reports. In this way, data are always available with 
respect to what each person intends to do, what he/she 
actually did, and a written rationale for changes in 
plans that became necessary as implementation activitie 
were initiated. Dr. Rapley reported that the "change 
reportis" were especially valuable in helping in further 
rrr.ceptualization of career education goals and objec- 
tives. 

In this project, a ver\' great deal of attention is 
being paid to teacher activit:ies in the classroom re- 
lated to career education. They are in the process of 
categorizing these activities in accordance with the 
Louisville Career Education Model. To be able to 
accurately categorize such teacher activities in a 
reliable manner is one of the biggest problems to be 
solved in this evaluation effort. A considerable 
degree of attention is being devoted, in the total 
process evaluation effort, to (a) documenting the kind 
and degree of in-service education given teachers in 
career education; Cb) verifying that what was learned 
through in-service was zctuiciZtJ applied in the class- 
room; and (c) arranging vhat takes place in the 
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teachine/leaming process in some kind of conceptual 
hierarchy. The Jefterson County career education pro- 
ject is one that operates conceptually within the 
fra:nevork of the entire teachlng/learr Ing process — not 
as a set of isolated activities to be added to other 
thin«»s teachers do. It is an approach to process evalu- 
ation that appears to hold great promise. 

At the time of the "mini-conference" Dr. Rapley 
attended, major attention was being devoted to process 
evaluation. It should be noted, however, that in the 
design for this project, both a Director of Process 
Evaluation and a Director of Product Evaluation were 
appointed. By the time this monograph appears In 
print, efforts of the Director of Product Evaluation 
should also be available* 



DR. CARV JARMER 

Dr. Jarmer has devoted considerable time and effort 
to identifying, using, and building new kinds of instru- 
ments required for evaluation of career education. He 
provided seminar participants with short descriptions of 
three such instruments. 

The first, developed by Glen Rask and Arvin Bloom 
in Colorado, is called the "Teacher/Administrator 
Career Education Needs Assessment". Its basic purpose 
is to measure the extent to which teachers and adminis- 
trators understand basic career education concepts. 
This kind of assessment is obviously a prerequisite to 
the effective use of such concepts in the classroom. 
As such, it is one preliminary v^ay of verifying that a 
career education "treatment" can be applied. 

A second instrument, also developed by Rask and 
Bloom, is called the "Student Assessment Instrument^. 
(Its formal title may differ from this, but this was 
the way it was reported in the "mini-conference".) 
This is an instrument used for ptcdiict zvaluaiA^n in 
career education through direct administration to 
students. It measures such things as: (a) attitudes 
toward work; (b) self awareness; (c) knowledge of 
occupations; and (d) future plans. Dr. Janaer reported 
this, too, to be an instrument he has found to be of 
value to him. 
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A third measure, developed by Dr. Jarmer, is 
designed to measure relationships between parent unaer- 
stand-'ig and Involvement in career education. Dr. Jarmer 
has collected both reliability and validity data on this 
parent assessment instrument. 

It Is \pbvious that, in contrast to the Jefferson 
County, Ket^tucky effort, Dr, Jarmer, in Kansas, is con- 
centrating relatively more on assessing student career 
development\as opposed to the total teaching/learning 
process. These first two examples provide an interesting 
contrast of irurrent evaluation efforts and concerns in 
career education. 



DR. ELLEN MEISTER 

In Madison, Wisconsin, Dr. Meister is responsible 
for large parts of the accountability question for the 
public schools. One major part of her efforts has been 
devoted to answering the question **Vhat impact on 
teacher behavior in the classroom results from funds 
expended for in-service staff development?". Since, in 
Madison, somewhere between $20,000 to $70,000 per year 
has been expended for staff development activities in 
each of the last five years ^ this becomes a very impor- 
tant question — especially in times of financial crisis. 
She has developed, for this purpose, an inventory of the 
amount of in-service given teachers in career education, 
and collected self reports from teachers specifying 
those career education skills the teachers believe they 
have acquired as a result of this training* 

Dr. Meister has also developed a number of instru- 
n-ents and nacerials for use in helping career education 
practitioners make evaluative judgments. Already 
developed is a 70-item instrument, involving eight 
categories, for use in evaluating career education 
materials. At present, she is developing an evaluation 
kit for use by teachers in evaluating "homemade" career 
education materials and evaluation instruments. Further, 
she reported herself to be currently working on trying 
to collect instruments appropriate for use in measuring 
the impact of involvement of the business-labor-industry 
community on the effectiveness of the career education 
efforts in Madison. 
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She has developed and is using a rather unique 
approach to evaluating the quality of career education 
activities in the classroom. This procedure involves 
taking non-involved, non-trained teachers into class- 
roon.s where teachers who have received in-service staff 
development in career education are teaching. The non- 
involved, non-trained teachers are asked to identify 
career education goals and evidences of success in 
reaching those goals, after spending from half a day to 
one and one-half observing in classrooms. The biggest 
practical problem to be solved here is to know ichzn to 
send the ''non- involved" teachers into a given classroom 
because, of course, the "involved" teachers are not 
using career education activities every hour of the 
day — nor even every day. 



HOVARD HEITZEG 

Mr. Heitneg has played a key role in evaluation of 
career education efforts in Pontiac , Michigan, In 
Pontiac, a najor atten:pt has been made to train teachers 
counselors, and others in career education methods and 
procedures using as a basis Robert Karkhuff's approach 
to interpersonal skills development, 

In Pontiac, considerable use is made of simulation 
activities in carrying out evaluation of career educa- 
tion. As one example, using the Karkhuff approach, 
students were encouraged to develop action plans for 
getting a job. They were j^iven both a ''people" job and 
a "thing" job for which to apply and told they would be 
paid S5 if, after the job inter^ciew, they were offered 
the job. Using professional job interviewers, inter- 
views were held with 20 students — ten of whom had been 
exposed to career education and ten who had not. The 
interviewers, of course, were not told which were the 
"career ed" students and which were not* Instead, they 
were told to hire only ten of the 20 students who 
applied. Of the ten hired, nine had come from the 
''career ed" group of students. This was presented as 
behavioral evidence that career education did, indeed, 
>E:ssist students in the acquisition of "job-getting" 
skills. 

V ^ 
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Mr. Heltzeg is convinced that ™ny area's of career 
education can be effectively evaluated bv \ise of simula- 
tion devices with random samples of "career ed" versus 
"non-career ed" students. He was currently working on 
such a simulation approach designed to test the degree 
to which career education efforts have improved student 
career dec Is Ion -making skills. The simulation approach 
to evaluation of career education, as It 13 being 
developed In Pontlac, Michigan, Is one that appears to 
hold high probabilities for effective replication else- 
where in the country — whether or not one chooses to use 
the Karkhuff approach. 



JOE GASTRIGHT 

In Cincinnati, the Ohio Model For Career Education 
has been introduced. This model contains three specified 
student outcomes for career education at each grade level. 
After teachers have received in-service staff development 
and have been given curriculum guides for use in applying 
this model, Mr. Gastright and his staff are charged with 
going into classrooms and collecting data that will 
answer the question "What percent of teachers are 
engaging in activities aimed at the three student out- 
comes?". 

In order to carry out some fom of product evaluation, 
the evaluation staff of the Cincinnati Public Schools 
asked teachers at each grade level to construct five 
items that would reflect what students should know as a 
result of teachi^r actions in career education. While 
unable to obtain suitable items in sufficient quantity 
at the elementary school level, they were able to con- 
struct a 60-lten instrur^ent for use at the junior high 
school level. Unfortunately, after the total Instrument 
was constructed, they could not find teachers agreeing 
that the items, in fact, represented what the teachers 
were trying to do. It was thus not surprising when, 
after the instrument had been administered to ''career ed" 
and to "non-career ed" students, no statistically signi- 
ficant differences in test scores were fotmd. 

One significant benefit of this approach is that it 
allcrwed a detemlnation to be r^de regarding what 
teachers, in operational terns, thought they were 
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supposed to be doing in career education. They dis- 
covered that, by and large, teachers appeared to think 
the task was primarily one of teaching the content of 
occupations (e*g., "What does a car;: ographer do?" was a 
test item made by one teacher), rather than broader, 
lifestyle considerations related to careers. (Note: Of 
course, this could have been due to the simple fact that 
such items are easier to construct than are those in- 
volving broader dimensions of career education.) 

One very significant career education evaluation 
activity nov underway in Cincinnati involves the testing 
and validation of a ten-point rating scale for use by 
elementary school pupils which they constructed arciand 
the theme "Wr-y do people work?". This scale was based 
on three broad areas of work values: (a) societal 
reasons ; (b) personal sat is fact ion ; and (c) economic 
need. They have found, in administering this Instrument 
to elementary school pupils, large differences in student 
responses. In general, responses went from a primary 
enrphasis on "economic need" in the First Grade, to an 
e^Tphasis on "personal satisfaction" in the Sixth Grade. 
This instrument, and its conceptual base, deserve 
careful study by those concerned with evaluation of 
career education. 



1}?,. ELVIS ARTZREURY 

As Direct'^r of the Partners in Career Education Pro- 
ject in Arlington, Texas, Dr. Arterbury has launched and 
is operating a major effort to develop and validate 
instruments for administration to students related to 
each of the nine "learner outcor'>;" found in the OE 
policy paper. An Jntrcdncticn T. Jarcct Educc.ticn, This 
project began by asking a vide range of individuals 
representing both educators and the general public, the 
question "Vhat should the puhlic schools be doing?". 
After a great deal of refinement, the responses were re- 
duced to a set of 177 basic learner outcomes for career 
education. Vcrking with Vestinghouse Learning Corporation, 
they have nov developed assessment measures for 80 of 
these 177 basic learner outcomes. In doing so, they have 
found it necessary to construct three items for each 
learner outcore, and to make each item one with a four- 
choice response format. 
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A total of 273 test items were developed for use In 
the Texas Career Education Measurement System (CEMS). 
These items were organized into 16 instruments for the 
measurement of 63 of the 80 basic learner outcomes and 
then packaged into the measurement system. Further, 
they have now developed one survey test built on items 
taken from the 16 category tests which covers nine 
broad categories. This one 45-item survey test is 
currently being used throughout Texas for purposes of 
conducting a Statewide needs assessment for career 
educat ion . 

It seems safe to say that the Partners in Career 
Education Project has done more to develop items and 
measures for use with each of the nine basic learner 
outcomes represented in A)! Jnt\cducticn To Caizcx 
FducatA.cn than any other single project in the United 
States. The primary approach being used is that of 
product evaluation. It is a system deserving of ver>* 
careful study by all interested in the evaluation of 
care*!r education. 



DR. CARROLL CURT IS 

Dr. Curtis reported that, in Pennsylvania, the 
creation of Career Resource Centers (for use by high 
school students from a given geographic area in career 
exploration and career decision-r-aking) has helped 
greatly in encouraging local school systetrs to begin the 
implementation of career education. Students using such 
centers are asked to rate the center on helpfulness to 
the student. These student ratings have, almost without 
exception, been consistently high. This has apparently 
encouraged many areas in Pennsylvania to construct such 
centers . 

Dr. Curtis also pointed out three current operational 
restrictions that are currently hindering compiehensive 
career education evaluation efforts in Pennsylvania. 
First, he pointed out the increased legal restrictions 
now existing with reference to any form of attitudinal 
testing. Second, he indicated that evaluation of career 
education has been hindered by the use of goals that are 
too global in nature — (e.g., the use of ''Job Placement" 
as a criterion for evaluating career education is of 
doubtful validity in view of the cany factors affectir.g 
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success In this area) — and pleaded for use of much more 
specific evaluative criteria. Finally, he questioned, 
from a conceptual point of view, use of such criteria 
as "career stability'' in the evaluation of career 
education, in light of the fact that, for many stud-^nts^ 
to change occupations is "good", not "bad". (Note: While 
these appear to be problems in Pennsylvania, it should hm 
noted that the two examples of criterion problems men- 
tioned by Dr. Curtis are net included among the nine 
learner outcomes listed in the official OE policy paper 
on Career Education.) 



DR. JANTT SHELVER 

Since 1971, Dr. Shelver has been operating a program 
called ^'Career Awareness For Secondary and Elementary 
Students" (CASES) in Sioux Falls, South Dakota. This 
is a program built on a joint concern for self awareness 
and occupational avareness. To measure <5ei|5 OiOaTCnC-A^ , 
at the K-3 level, she has used the "Pictorial Self 
Concept Scale" — a 50-card deck cf pictures which pupils 
are asked to put in thr.-e stacks: (a^ "like me"; (b) 
"sonerimes like me"; and (c) "not like me". At the 
Gradct 4-6 level, she uses Coopersmith * s "Self Esteem 
Inventory" to measure self awareness. 

To measure czcu]Xitlcizal Jica^tztic^t ^ Dr. Shelver uses 
14 slides of workers in various occupations, each 
representing one of the 15 CE clusters. (Marine Science 
was eliminated because no one in Sioux Falls is errployed 
in that cluster.) Each slide depicted a worker in some 
occupation in Sioux Falls. As students are shown the 
14 slides, they are asked to answer the following 
questions: (a) '*Vho am I?"; (b) 'Vhat do I do?"; and 
(c) "Vould I like this job?". 

Using these instruments. Dr. Sheiver found that her 
career education efforts did ser\*e to reduce occupatiDnal 
sex stereotyping among female (but not among male) pupils 
at the K-3 level. Further, she found that the C^SES 
career education approach did improve self concept of 
pupils at the Grades 4-6 level. Vhile these st^udent 
cutccnes are, to be sure, only a ver>" narrow segment cf 
vhat would be considered to be a comprehensive career 
education effort, the positive results Dr. Shelver 
obtained are most encouraging. 
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DR, ARTHUR BERRY 



In the State of Maine, two major efforts have been 
initiated aimed at initial evaluation of career educa- 
tion efforts. One of these, involving the use of 
pre-tests and post-tests of occupational Informatiim and 
terminology with teachers and counselors, was conducted 
as part of a National Alliance of Businessni^r* Career 
Guidance Institute, Results were very positive in terms 
of increases in teacher knowledge and perceptions of the 
occupational world. 

The second major evaluation project Dr* Berry reported 
on is being conducted for Project REVAMP— a Part D, VEA 
career education project in South Portland, Maine* 
Project REVAMP operates under an assumption that career 
education is a concept to be infused into existing 
instructional programs, not a. new kind of program to be 
added to existing ones. Accordingly, evaluation proce- 
dures are designed to measure the extent to which 
currently employed teachers, counselors, and curriculum 
resource persons incorporated career education activities 
into their regular activities during the school year. To 
accomplish this objective, the evaluation team (one from 
curriculum, one from guidance, and one from elementary 
education) meets with local career education persons at 
the beginning of each year and helps them determine 
specific goals for the year. At the end of the year, 
they meet again to determine the extent to which the 
goals were met. 

A significant part of this evaluation effort is con- 
cerned with process evaluation. For example, evaluation 
reports include statements such as: (a) "150 pieces of 
career materials were distributed to pupils"; (b) "220 
teachers received in-service staff development in career 
education"; and (c) "40 career interest and aptitude 
tests were given to high school students". A plan for 
product evaluation is also included in the South Portland 
total career education evaluation design. 



DR. THOMAS SMITH 

Career education activities in Covina Valley, 
California, have been on-going since 1971 when 12 
teachers expressed interest in restructuring their 
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courses using career education learning packages made by 
each teacher In an In-cervlce career education workshop. 
At present, about 50% of all Covlna Valley teachers are 
providing some kind of career education activities in 
their classes. Because each teacher is encouraged to 
'*do his/her thing" in Inventing career education in the 
classroom, it is almost Impossible to evaluate as a 
total career education effort. Still, the rapid 
growth in the number of teachers taking this approach, 
and their continuing interest in doing so, carry posi- 
tive connotations in spite of the absence of hard 
product data. 

As a result. Dr. Smith and his colleagues developed 
a student instrument called the "Career Development 
Status Survey", consisting of 50 items and based on the 
13 basic career education concepts developed by the 
California Career Education Task Force. In applying 
this instrument, they found significant differences even 
among students who had been exposed to career education 
activities by different teachers, as well as differences 
between "exposed" and "non-exposed" students. 

Further evaluation efforts have been made with respect 
to the career guidance component of career education. 
One example was a study using 8th Grade pupils who were 
exposed to a very intensive guidance effort aimed at 
helping them make curricular and career plans. Data 
from a previous year*s class of 8th graders showed 50% 
having "no plans" when in the 9th Grade. After this 
effort, only 2% of these students said they had "no 
plans" when in the 9th Grade. Another approach they have 
used is one of determining tl extent to which ACT Career 
Planning Profile results coincide with the actual kinds 
of course enrollments made by students once they enter 
community college programs. 

Dr. Smith outlined eight areas he considers appro- 
priate for use in evaluating career education including: 
(a) results of incremental improvement efforts of 
teachers to infuse a career emphasis in the teaching/ 
learning process; (b) Student knowledge of occupations; 
(c) Student knowledge of available postsecondary educa- 
tional preparation programs; (d) Student and teacher 
knowledge of basic career development principles; (e) 
Student knowledge of reality factors impinging on full 
freedom of occupational choice (e.g., geographic, 
financial, etc.); (f) Student skills in the 
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decisions-making process; (g) Student knowledge of 
follov«-up results from past classes of students; and 
(h) "Customer satisfaction" results from students and 
the various kinds of Individuals responsible for Imple- 
menting career education. 



DR. RICHARD RUFF 

Now In their fifth year of State-legislated funding 
for career education, the Arizona State Department of 
Education has developed an elaborate evaluation system 
for use in assessing their career education efforts* 
These Include two 45->ltem student tests (one for Grades 
3-7 and the other for Grades 8-12) containing Items for 
each cell In the Arizona career education matrix. Using 
these product evaluation Instruments, striking differences 
have been found between "career ed" as opposed to "non- 
career ed" students. As an example of the kinds of Items 
used In the area of "self awareness", students are asked: 
(a) "How sure are you that you can do what you want to 
do?" and (b) "How sure are you of your occupational 
decisions?". 

In addition to these product evaluation measures, 
Arizona also uses a process evaluation approach on an 
annual basis. Having such data available each j^ear over 
a number of years has enabled them to say to members of 
the Arizona State Legislature such things as: "Over the 
last two years, we have had a 24% Increase In parental 
Involvement In career education". Dr. Ruff stressed the 
Importance of collecting these kinds of data on an annual 
basis so that Incremental Improvement can be shown with 
process evaluation approaches to career education* 

Arizona also uses a High School Student Follow-up 
Questionnaire with students one year out of high school. 
This Instrument contains Items such as: "Did your high 
school prepare you to (a) get a job, (b) prepare for a 
job", etc. Data from this follow-up Instrument are 
being used to show the general public that career educa- 
tion works ^ght nou) In Arizona. Apparently, these data 
have been most effective In convincing people of the 
worth of career education. 

Arizona, like Colorado and Kansas, also has developed 
a test to measure teacher understanding of basic career 
education concepts. 

26 



Over the last several years, career education curricu- 
lum units have been developed by Arizona classroom 
teachers, validated by career education specialists In the 
Arizona State Department of Education, and then distribu- 
ted to teachers throughout the State. Validation data, 
for each such unit, now Includes answers to questions such 
as: (a) To what extent do pupils attain the unit objec- 
tives; (b) Do teachers enjoy using this unit?; (c) Do 
pupils enjoy the unit?; and (d) What costs are Involved 
In using the unit?- If one were looking for a single 
State Department of Education where career education 
evaluation has been systematically and carefully studied, 
It would be hard to find a better example than the Arizona 
State Department of Education. 



DR. PHILLIP SPTETH 

The Dade County (Miami, Florida) career education 
effort operates with a number of components developed at 
the county office level- Each school Is free to pick 
those components (if any) that they want to use In their 
career education activities. Thus, each school has, in 
a sense, a unique program. Partly for this reason, 
evaluation plans for career education call for selecting 
both experimental and control pupils from within the 
same school — not in comparing pupils in one school with 
another. 

Considerable attention has been devoted, especially 
at the elementary school level, to assessing the effects 
of a career education emphasis on improvement in academic 
achievement of pupils. To do this, they have devised a 
complicated system for classifying pupils into '^clusters" 
— (e.g., white, low ability, etc.) — that will allow them 
to determine, for any pupil by the end of the school 
year, whether that pupil is doing (a) better than expec- 
ted, (b) about as expected, or (c) worse than expected. 

Perhaps because of the wide variability in specific 
career education practices adopted by the various 
schools in the county, mixed results have been obtained 
in evaluations of career education. In some schools, 
pupils appear to have gained significantly in academic 
achievement, while in other schools they have not. The 
only apparently consistent finding that appears to hold 
up, county-wide, is that pupils exposed to a career 
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education approach come to school nore frequently than 
do those who are not exposed to career education. (In 
one school comparison, they also found that, with a 
career education emphasis Introduced, t^choA attendance 
as well as pupil attendance Increased.) Another possible 
reason for the observed variability of results Is that, 
during the 1974-75 school year, the experimental treat- 
ment period was only six months. For 1975-76, the period 
la Intended to cover the entire school year. 



SlTiMARY OBSERVATIONS CF CURRENT PRACTICES 

The 12 career education evaluation "practitioners", 
whose activities have been summarized in this section, 
obviously vary greatly both in their approaches to 
evaluation and in their concepts of career education. 
The mixture between an educational reform and an educa- 
tion in careers emphasis is very obvious. So, too, is 
the mixture between process and product evaluation. 
Those who appear to have adopted a concept of educational 
reform as the basic goal of career education seem to tend 
much more toward Initial use of process evaluation mea- 
sures, with product evaluation measures coming along 
later. On the other hand, those whose perceptions of 
career education appear to center more directly on 
career development of Individual students seem to go 
much more directly toward a product evaluation approach. 

In all cases, these 12 examples have, hopefully. 
Illustrated that, in spite of the problems outlined In 
the first portion of this monograph^ career education 
evaluation attempts aTC currently being carried out in 
areas of the country that cover every major geographic 
region — from coast to coast and from North to South* 
The problems Involved in the evaluation process have 
not stopped those convinced of the need to provide data 
concerning the worth of career education. 



USE OF OE EDUCATION LEARNER OI'TCOMES 
IN EVALUATION EFFORTS 

Both the problems involved in and the need for process 
evaluation of career education were stressed in the first 
portion of this monograph. The emphasis on product 
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evaluation found In this section is in no way Intended 
to demean nor to detract from the absolute necessity for 
process evaluation efforts. Rather, It Is core correctly 
Intended to etnphasize what Is regarded as an absolute 
necessity If cateer education Is to continue to be pic- 
tured as a concept rather than as a program. With the 
active Involvement of many kinds of persons from many 
segments of the total community in career education, it 
is essential that those concerned with evaluation of 
career education concentrate significant attention, at 
this period in the short history of this evolving con- 
cept, on assessing student benefits to be derived from 
a comprehensive career education effort. Such an 
approach, while obviously ignoring the differential 
contributions made by various "actors'* in the effort, 
is essential for answering the question of whether or 
not career education can produce some demonstrated 
benefits for students. If this can be done, career 
education can, in future years, zero in on more 
sophisticated and more finely-tuned evaluation proce- 
dures. Unless this is done, it is possible that career 
education will not have a long future. 

In the OE policy paper, An JntAoduct^cn To Ca*ieQA 
EdiiCatAcn^ nine learner outcomes, stated essentially in 
product evaluation form, are listed. This entire list 
is preceded, in the OE policy ppper, by an introductory 
phrase which says: 

'XaxecA education Accfci tc pncduce lnd<v<dua(!6 tdic, 
'xhcn thzi^ ^cav'C teheed [at an;v ag^ cx at anu Icvzf ) 

After a Ti sting of the nine learner outcomes, the fol- 
lov^;i^ Paragraph appears in the OE p^lJcy paper: 

"It is important to note that these learner goals are 
intended to apply to persons leaving the formal educa- 
tional 4(/6tCJ^ for the world of work They are not 
intended to be applicable whenever the person leaves 
a particular school. For some persons, then, these 
goals become applicable when they leave the secondary 
school. For others, it will be when they have left 
post-high school occupational education programs. For 
still others, these «oals need not be applied, in toto, 
until they have left a college or university setting. 
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'*Thus, the applicability of these learner outcome 
goals will vary from individual to individual as well 
as from one level of education to another. This is 
consistent with the developmental natuj^ » and the 
basic assumption of individual differences. Inherent 
in the concept of career education." 

Given both the predlcatory phrase and the paragraph 
that follows the listing of the nine OE learner outcome 
goals, both sets of consultants at the two "mini- 
conferences" concerned with evaluation of career educa- 
tion were asked to comment on the suitability of the 
goal and our current readiness for using it in the pro- 
duct evaluation of career education. Valuable recommen- 
dations (sometimes with consensus but often without) were 
received from individual members of both "mini-conferences" 
Participants in the second of these "mini-conferences", in 
addition to commenting on the applicability of the nine 
learner outcome goals, suggested revisions in several of 
them (over and beyond revisions that had already been 
made by the Director, Office of Career Education), 
Moreover, they suggested one additional learner outcome 
goal to be added to the original list of nine. 

In this section, then, each of the now tzn OE learner 
goals, in {^onm tkzij oAz yioik^ Intzndzd to be ^txtzd^ 
will be delineated, and consultant recommendations con- 
cerning our readiness for using each goal will bt summa- 
rized. 



GOAL 1: COMp_ETE!rr IN THE BASIC ACADEMIC SKILLS REQUIRED 
FOR ADAPTABILITY IN OUR RAPIDLY CHANGING SOCIETY 

A clear majority, but not all, of the consultants con- 
sidered this to be a reasonable learner outcome goal for 
career education. In the view of OE's Office of Career 
Education, this is the single most Important learner out- 
come goal for career education, as a vehicle for educa- 
tional KZ^o^. A proposal for zducdtionai. KZ^^cnm that 
Ignores this learner outcome goal cannot, in the Office 
of Career Education's (OCE's) opinion, be justified 
at all. 

While most of the "mini-conference" partlclpt?nts 
agreed on the appropriateness of this learner outcome 
goal, they disagreed on the kind of measures appropriate 
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for use in measuring progress toward this goal. In the 
first of these two "mini-conferences", consensus seemed 
to exist that norm-referenced tests are what the general 
public wants and, thus, what should be used. A signifi- 
cant minority of these participants, however, argued 
strongly for use of criterion-referenced achievement 
tests. General agreement was present that norm-referenced 
achievement tests have serious limitations and prooably 
will be replaced with other approaches to measuring 
academic achievement, but most felt them to be more 
appropriate than criterion-referenced tests at the 
present time . 

In the second of these "mini-conferences" on evalua- 
tion of career education, most of the discussion centered 
around the potential some of the newer approaches to 
measuring ^uncXiona^, iitOA^itij have for assessing this 
learner outcome goal. It was pointed out that several 
significant beginnings have been made in developing and 
in validating functional literacy measures. Most commonly 
mentioned were the functional literacy measures developed 
as part of the National Assessment of Educational Progress 
project of the Education Commission of the States. The 
recently much-publicized APL tests of functional literacy 
from the University of Texas were also discussed as 
promising. So, too, was a recent monograph by Dr. Thomas 
Sticht of HUMRO which contains a good working definition 
of functional literacy. Finally, the NIE Model IV 
(Mountain Plains) project was mentioned as one now 
developing functional literacy measures. A general con- 
sensus seemed to exist, in this second "mini-conference" 
that, while standardized norm-referenced achievement 
tests may be suitable for use in assessing this learner 
outcome goal for elementary school pupils, a measure of 
functional literacy would be preferable at the junior 
and senior high school levels. 

In the view of OE's Office of Career Education, the 
use of norm-referenced achievement tests are appropriate, 
at this point in time, for use nt both the elementary and 
secondary school levels in assessing this learner outcome 
goal. In addition, at the secondary school level, use of 
some kind of functional literacy measure should be 
encouraged. Use of functional literacy measures, as a 
replacement for norm-referenced achievement tests at the 
secondary school level, will become possible only when 
academic teachers at that level accept such measures as 
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valid Indicators of what they are teaching. It appears 
that this will not happen in the immediate future. 



GOAL 2: EQUIPPED UITH GOOD VOKK HABITS 

Both sets of "mini-conference" participants agreed 
that this learner outcome goal is both appropriate and 
measurable. However, disagreement and controversy was 
found on many of the technical issues surrounding its 
use. For example, some participants agreed that it is 
feasible to consider measuring knowlzdgz of good work 
habits, but felt that they could not be measured 
bziiCLvloHjoZZy. Others contended that behavioral measures 
of work habits are available. Some questioned the use 
of teacher checklists as indicators of pupil work habits, 
but others pointed out that, when such checklists are 
■used with Kindergarten pupils, the results correlate 
positively with reading score gains in Grade 1. 

One person emphasized the fact that, with current 
restrictions existing on using any form of attitudinal 
measure as part of the pupil's grade, the potential for 
marking students on "work habits^' may be questioned 
severely in some quarters. 

Another very thorny problem was raised by one partici- 
pant who asked what differences exist between what is 
known as "work habits" and the "work ethic". General 
agreement seemed to be present that these tr/o terms, 
while not unrelated, should be kept as separate as 
possible in developing assessment measures. This, of 
course, can be most easily and clearly done if measures 
of "work habits" are developed in behavioral terms and 
measures of the "work ethic" are developed in attitudinal 
terms. (Note: The problem of distinguishing between these 
two terms is a conceptual one that could be debated at 
some length. It is not considered appropriate to include 
such a debate as part of this monograph.) 

In spite of these several areas of disagreement and 
controversy, a clear consensus was present, among both 
sets of "mini-conference" participants, that this 
learner outcome goal is an appropriate one for use and 
that it can be measured in a satisfactory manner. 
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GOAL 3: EQUIPPED WITH A PERSONALLY MEANINGFUL SET OF 
WORK VALUES THAT FOSTER IN THEM A DESIRE TO WORK 
(Note: Revised from earlier OE statement) 

This learner outcome goal, as originally stated In 
the OE policy paper on career education, had Included 
the phrase "capable of choosing". Participants In both 
"mini-conferences" agreed that, at best, this is a phrase 
difficult to define In measurable terms. Several felt 
this phrase to be a meaningless one. 

It was suggested that, In evaluating the degree to 
which this goal has been attained, evaluators seek 
measurements at the following three levels: (a) the num- 
ber of kinds of work values of which the student is 
aware; (b) the definiteness with which the student has 
decided which work values are important to him/her; and 
(c) the behaviors the student has taken that demonstrate 
possession of one or more of his/her work values. Con- 
sensus seemed to be present that, at each of these 
three levels, suitable measures could be constructed. 
One seminar participant suggested that, perhaps, a 
simple process measure such as the number of teachers 
who are trying to reach this objective might be suffi- 
cient for use, but this suggestion was not apparently 
accepted by most of the participants. 

Several "mini-conference" participants reported they 
are already collecting data related to this learner 
outcome goal. Joe Gastright reported having data for 
3rd and 6th graders around the question "Why do people 
work?" that produced dramatic results. Janet Shelver 
reported both 1st and 6th grade pupils in Sioux Falls 
seemed to pick up the concept of work values quickly and 
easily. Dick Ruff r«?ported having data showing that 
elementary school pupils exposed to career education 
spend more time on homework than do non-career ed pupils 
(self report data). 

It may well be that, for this learner outcome goal, 
two different kinds of evaluation measures may be re- 
quired. As one participant pointed out, one can measure 
whether or not pupils possess work values. One can also 
measure pupil desire to work. However, one cannot 
measure directly the ^zlatloMhip between the two. This 
seems to be a very valid point and well worth considera- 
tion. 
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GOAL 4: EQUIPPED WITH CAREER_DECISIONMAKING SKILLS, 
JOB-HUNTING SKILLS.^ AND JOB-GETTING SKILLS 



There was almost unanimous consensus, among partici- 
pants in both of the ''minl-conf erences** , that this 
learner outcome goal could be assessed. Consensus seemed 
particularly strong if the measures used consist of 
instruments assessing knotA:le,dgz of such skills rather 
than behavioral demonstrations of their uses. There also 
seemed to be consensus that, even if direct behavioral 
assessments based on routine actions could not be assessed, 
this learner outcome goal lendu itself very well to 
assessment through simulation procedures which are be- 
haviorally oriented. Both Dick Ruff and Elvis Arterbury 
reported success in measuring this goal through knowledge 
tests. 

Other possible instruments for use in measuring this 
learner outcome goal were suggested by several partici- 
pants. For example, Houghton-Mifflin's A66U6rmnt of^ 
CoAZCA Vzvzlcpmcnt has d section on *'work skills*' that 
could be useful here. Dr. Robert Campbell, Center for 
Vocational Education, Ohio State University, was reported 
to have a "job-coping skills test" which would be suit- 
able for assessing the variable of "job-holding" skills. 
Both "job-seeking" and "job-getting" tests were reported 
to be available from the Center for Research in Vocational 
Education, the University of Wisconsin. Several partici- 
pants recommended examining measures in all of these areas 
now available from the National Assessment of Educational 
Progress project conducted by the Education Commission of 
the States. 



GOAL 5: EQUIPPED WITH JOB SPECIFIC OCCUPATIONAL SKILLS 
AND INT E RPERSONAL SKILLS AT A LEVEL THAT WILL ALLOW THEM 
TO GAIN ENTRY lOTO AND ATTAIN A DEGREE OF SUCCESS IN THE 
OCCUPATIONAL SOCIETY (Note: Revised from earlier OE 
statement) 

The original OE statement of this goal, as it appears 
in the OE policy paper, contained a typographical error 
in that the word "personal" was used rather than the 
word "interpersonal" which was intended. In addition to 
correcting this misprir.t, participants suggested substi- 
tuting the phrase "job rfpecific occupational" for the 



word '"vocational" so that clear and unambiguous refer- 
ence is made to the world of paid employment — (i.e., the 
generic tern "vocational** can be interpreted to include 
unpaid, as veil as paid, work). With these two changes, 
participants in both "nini-conf erences*' agreed tliat this 
learner outcome goal is suitable for use and capable of 
being measured. 

The prime reservation, insofar as the measurement 
problem is concerned, was expressed in terms of pointing 
out that an accurate assessment of "job specific occupa- 
tional skills" would require literally thousands of 
specific measures — both paper and pencil tests and per- 
formance measures. Others pointed out that, if one 
limits his/her thinking to the job entry level, the 
problem may not be as complicated as it appears to be at 
first glance. 

One participant suggested that tlie term "interpersonal 
skills" be removed from this learner outcome goal and be 
made a part of Goal 4, No strong consensus appeared to 
be present cn this point. There did seem to be consensus 
that the term "interpersonal skills" should be retained 
somewhere among the learner outcome goals- 

Participants in the first of these two "mlni-conf erenccfi 
agreed tliat student follow-up data would constitute appro- 
priate data for use in assessing this learner outcome goal. 
However, participants in the other "mini-conference" dis- 
agreed with this suggestion, OF/s Office of Career Educa- 
tion believes that, while follow-up data constitute one 
form of acceptable data for this learner outcome goal, 
they will not be sufficient in and of themselves. 



GOAL 6: EQUIPPED WITH A DEGREE OF SELF-UNDERSTANDING AND 
inrPER STANDING OF EDUCATIONAL-VOCATIONAL OPPORTUNITIES 
SUFFICIENT FOR MAKING SOUND CAREER DECISIONS 
(Note: Revised from earlier OE statement) 

As originally written in the OE policy paper, this 
learner goal assumed a direct causal relationship between 
stuuent self-understandings, understandings of educational- 
vocational opportunities, and actual career decisions made 
by students. Participants seemed to agree that the 
learner outcome goal would be more susceptible to accurate 
measure if it concentrated on possession of skills required 
for making sound career decisions without carrying any 
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necessary implications for making judgments regarding 
the soundness of those career decisions any given 
student finally makes. Taat is, the eiiiphasis should be 
on whether the decision is ^'reasoned", not on whether 
it is "reasonable" (as Judged by others). The revised 
goal stat^rr»c?ut appearing above appears to meet this 
concern. 

One part ic ipant suggested thau the term "..at eer 
planning skills" could be substituted for the longer 
phrase used in this learner outcone goal and that > if 
this is done, that term could be added to others already 
present in Learner Outcome Coal A, thus making Learner 
Outcome Goal 6 unnecessary. There did not seem to be 
any strong agreement with this position. Thus, thau 
recommendation was not followed. 

GOAL 7: AVARE OF MEANS AVAILABLE TO THEM FOR COKTT!;UINC 
AND RECURKENT EDUCATION (Note: Revised from earlier OE 
statement) 

Participants suggested eliminating the phrase "once 
they have left the formal system of schooling" from this 
goal statement in that it is unnecessary, given the pre- 
dicatory statement that precedes the list of learner out- 
come goals. This seemed to be a good suggestion and it 
v;as followed. 

Several participants pointed out the necessity for 
collecting data, with reference to this learner outcome 
goal, primarily in terms of the immediate future faced 
by students as they are about to exit from the formal 
school setting. That is, there will be no good way of 
assessing the degree to which they will be aware of such 
means several years after having left. Others disagreed 
with this suggestion and recommended that, if "means" 
are thought of as "skills", they can be assessed through 
more kinds of measures than simple tests of information. 

No participant, in either "mini-conference", indicated 
this learner outcome goal to be inappropriate nor inca- 
pable of measurement. 
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GOAL 8: EITHER PLACED OR ACTIVELY SEEKING PLACEMENT IN A 
PAID OCCUPATION. IN FURTHER EDUCATION. OR IN A VOCATION 
CONSISTENT VTITH THEIR CURRENT CAREER DECISIONS 
(Note: Revised from earlier CE statement) 

General consensus, at least among the second group of 
"mini-conference" participants, appeared to be present 
that, as originally written, there was no way in which 
any student could fail to meet it — thus making the goal, 
as a criterion measure, operationally useless. Others 
pointed out that, unless the words "actively seeking" 
were inserted into the goal statement, too many students 
would be sure not to meet it I The revised learner goal 
statement, as it appears above, represents a compromise 
between these two expressed concerns. 

Participants in the first of these two "mini-confer- 
ences'' agreed that student follow-up studies would con- 
stitute an appropriate measurement approach to use in 
assessing this learner outcome e,oal. Participant? in 
the scconi "TT.Jni-conference" did n^.^t disagree with this 
recotarwf ndat iou • 



GOAL 9: ACTIVELX SEEKING T ^ TIND MEANING AND MEANING- 
HXNESS THROU GH '> ^mK IK PRODUCTIVE USE OF LEISURE TIME 
(Note: Revised from earlier OE statement) 

As originally written in the OE policy paper, parti- 
cipants in the first of the two "mini-conferences" 
agreed that the goal was incapable of being measured. 
1 rior to the second of these "mini-conferences", the 
Director of the Office of Career Education in OE revised 
the goal statement as it appears above. With this 
revised goal statement, participants in the second 
"mini-conference'* seemed to agree that the goal was both 
reasonable and capable of being measured. 



GOAL 10: AWARE OF MEANS AVAILABLE TO THEMSELVES FOR 
CHANGING CAREER OPTIONS-^F SOCIETAL AND PERSONAL CON- 
STRAINTS IMPINGING ON CAREER ALTERNATIVES (Note: This 
is a new goal and does not appear in the OE policy paper) 

The suggestion for addition of this new learner outcome 
go^l came from Dr. Anita Mitchell, a participant in the 
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second of these "mlnl-conf erences'\ Dr. Mitchell pointed 
out that, as written, the set of OE learner outcome goals 
are directed primarily toward helping individuals adjust 
to society rather than showing them ways of adjusting 
society to themselves. Other participants in this 
"mini-conference" seemed to be in strong agreement with 
Dr. Mitchell on this point. So, too, was the Director 
of the Office of Career Education, 

The topic of how to measure this learner outcome goal 
was not discussed. It would not seem unreasonable to 
assume that it could be treasured, at least on a knowledge 
of information type inst:: ir^ent. As with several of the 
others, it would be difficult to measure in behavioral 
terms unless broken down into smaller components. 

In suasaary, it can be seen that, of the original nine 
learner outcome goals found in the OE policy paper, six 
have been revised as a result of the consultative assist- 
ance provided by these two sets of "mini-conference" 
participants. In addition, one new goal has been added 
to the list making ten in all. Those persons currently 
using the OE policy papier, An JnVicductLon To CaAZ(L.X 
EdiiCCiJticK^ as a source of learner outcome goals, are 
encouraged to substitute the revised set as presented 
here . 



CONCLUDING REMARKS 

Probleris associated vith evaluation of career educa- 
tion are complicated by the developmental narare of the 
career education concept, by the newness of the concept 
and its introduction in the nidst of a strong call for 
accountability in all of education, and, most of all, by 
the fact that career education seeks to remain a concept 
to be Infused into all existing educational programs 
rather than a new and separate program specialty in 
education. 

In spite of these difficulties, systematic and con- 
scientious attempts at evaluation of career education are 
currently being conducted in communities in all major 
geographic sections of the United States. While wide 
variation obviously exists in both the methodologies 
baing employed and in the ways in is:hicb career education 
is being conceptualized, the need and the desire to 
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engage in careful and systreit»atic evaluation procedures 
is evident. So, too, are initial signs pertaining to 
the efficacy of career education. 

Both process and product evaluation are needed in 
career education. Process evaluation will, for some 
time, be essential in assessing the degree to which 
career education is, in fact, serving as a vehicle for 
reform of the educational system. Product evaluation, 
based on a clearly defined set of learner outcome goals, 
will be essential in assessing the extent to which the 
career education effort is providing direct benefits to 
students. The revised OE list of "learner outcome 
goals" for use in evaluating career education efforts 
appears to be acceptable to those experts in educational 
evaluation who participated in the two OE "mini-confer- 
ences" on which this monograph is based. Further 
revisions will undoubtedly come in the years ahead - 

The need for careful attention to the problems and 
the potential inherent in evaluating career education 
efforts is obvious. That need is great at the present 
time and it is sure to increase in the future- Career 
education, no matter how great the conceptual and 
operational problems are, cannot avoid the necessity 
for careful and conscientious evaluation efforts. It is 
hoped that the contents of this monograph will encourage 
the further development and rapid proliferation of sound 
evaluation efforts among all those who profess to be 
engaged in career education. 
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APPENDIX A 

List of Mlnl-Conference Participants 
Evaluation of Career Education 
Conference #5 October 6-7, 1975 



Mr. Joe Gastrlght 
Program Evaluation Branch 
Cincinnati Public Schools 
Cincinnati, Ohio 

Mr. Howard Heltzeg 
Supervisor of Research 

and Evaluation 
Water ford, Michigan 

Dr. Carroll Curtis 
Director, RCD 
Pennsylvania State 

Departiaent of * -nation 
Harrlsburg, Penr - ^ -anla 

Dr. Gary Jarmcr 

Regional Career Education 

Project Director 
Colby, Kansas 

Dr. Richard Ruff 
Planning and Evaluation 
Arizona State Departiaent 

of Education 
Phoenix, Arizona 

Dr. Frank. P^pley 
Assistant Superintendent 

of Instruction 
Louisville Ptxbllc Schools 
Louisville, Kentucky 



Dr. Ellen Melster 
Madison Public Schools 
Madison, Wisconsin 



Dr. Thomas Smith 
Director, Research & PPS 
Covina Valley Schools 
Co^rlna, California 

Dr. Elvis Arterbury 
Partners in Career 

Education 
Arlington, Texas 



Ms. Janet Shelver 
Career Education Consultant 
Sioux Falls Public Schools 
Slorrsc Falls, South Dakota 

Dr* Arthur Berry 
Coordinator of Vocational 

Education 
University of Maine 
Gorham, Maine 

Dr. Phillip Spieth 
Director, Evaluation 

Studies 
Dade County Public Schools 
Miaai, Florida 
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APPENDIX B 



List of Mini-Conference Participants 
Validation Conference on 
Evaluation of Career Education 
Conference #19 March 2^ 1976 



Dr. Brandon Smith 
RCU Director 

Minnesota State Department 
of Education 

Dr. Anita Mitchell 
American Institute of 

Research 
Palo Alto, California 
(President, AMEG) 

Dr. Lois-ellin Datta 
Assistant Director 
Education and Work Group 
National Institute of 
Education 



Dr. Alice Scates 
Baireau of Planning and 

Evaluation 
U.S. Office of Education 

Dr. Conrad Katzenneyer 

Wisconsin Research and 
Development Center for 
Cognitive Learning 

University of Wisconsin 

(President, CEDAR) 



